AUC: A Better Measure than Accuracy in Comparing Learning Algorithms

Predictive accuracy has been widely used as the main criterion for comparing the predictive ability of classiication systems (such as C4.5, neural networks, and Naive Bayes). Most of these classiiers also produce probability estimations of the classiication, but they are completely ignored in the accuracy measure. This is often taken for granted because both training and testing sets only provide class labels. In this paper we establish rigourously that, even in this setting, the area under the ROC (Receiver Operating Characteristics) curve, or simply AUC, provides a better measure than accuracy. Our result is quite signii-cant for three reasons. First, we establish, for the rst time, rigourous criteria for comparing evaluation measures for learning algorithms. Second , it suggests that AUC should replace accuracy when measuring and comparing classiication systems. Third, our result also prompts us to re-evaluate many well-established conclusions based on accuracy in machine learning. For example, it is well accepted in the machine learning community that, in terms of predictive accuracy, Naive Bayes and decision trees are very similar. Using AUC, however, we show experimentally that Naive Bayes is signiicantly better than the decision-tree learning algorithms.